Large image databases 1 Running head: LARGE IMAGE DATABASES Exploring human cognition using large image databases

Most cognitive psychology experiments evaluate models of human cognition using a relatively small, well-controlled set of stimuli. This approach stands in contrast to current work in neuroscience, perception, and computer vision, which have begun to focus on using large databases of natural images. We argue that natural images provide a powerful tool for characterizing the statistical environment in which people operate, for better evaluating psychological theories, and for bringing the insights of cognitive science closer to real applications. We discuss how some of the challenges of using natural images as stimuli in experiments can be addressed through increased sample sizes, using representations from computer vision, and developing new experimental methods. Finally, we illustrate these points by summarizing recent work using large image databases to explore questions about human cognition in four different domains: modeling subjective randomness, defining a quantitative measure of representativeness, identifying prior knowledge used in word learning, and determining the structure of natural categories. Large image databases 3 Exploring human cognition using large image databases Over the last century, cognitive psychology has moved towards using more and more abstract stimuli in order to maximize experimental control. An example is the literature on category learning. Hull (1920), in his classic work exploring how people learn novel concepts, used Chinese characters as stimuli – a relatively naturalistic choice. In the 1950s, when the rigor of cognitive psychology was in question, Bruner, Goodnow, and Austin (1956) began studying concept learning using stimuli that had a very clear set of discrete features – the number, fill pattern, shape, and borders of a set of objects on a card. These abstract stimuli have become standard in category learning experiments (e.g., Shepard, Hovland, & Jenkins, 1961; Medin & Schaffer, 1978), with contemporary work using methods like multidimensional scaling to confirm the dimensions that people use to represent these stimuli (e.g., Nosofsky, 1987). Abstract stimuli support precision. Research on category learning, for example, has reached the point where it is possible to test the fine-grained predictions of a variety of detailed mathematical models of human behavior (for an overview, see Pothos & Wills, 2011). However, this precision comes at the potential cost of ecological validity: by using ever more abstract stimuli to improve the precision of our measurements, there’s the chance that the cognitive processes that we are measuring no longer correspond to the phenomena that we were originally interested in understanding. Do the same processes support learning abstract categories of geometric figures and the development of a child’s ability to discriminate dogs from cats? One approach to improving the ecological validity of cognitive psychology has been to use stimuli that allow people to make use of prior knowledge, making it possible to study the effects of this knowledge on category learning (Murphy & Medin, 1985). In this paper we highlight another axis along which methodological practices might change: the use of natural images, of the kind that can be found in large online image databases. This approach follows an emerging trend towards the use of natural images in research on computer vision (e.g., Deng et al., 2009; Torralba, Large image databases 4 Fergus, & Freeman, 2008), neuroscience (e.g., Simoncelli & Olshausen, 2001; Naselaris, Prenger, Kay, Oliver, & Gallant, 2009), perception (e.g., Geisler, Perry, Super, & Gallogly, 2001; Geisler, 2008), and visual cognition (e.g., Brady, Konkle, Alvarez, & Oliva, 2008). Large image databases have facilitated significant advances in these fields, bringing theories and empirical results into closer alignment with the problems people face in everyday life. Natural images can be used in cognitive science research in several different ways. In this paper, we explore these different uses of natural images, consider how some of the challenges of working with large image databases might be addressed, and use a series of case studies based on our own work to illustrate how these issues are negotiated in practice. When are natural images valuable? Natural images have been used in two ways in neuroscience and perception research: as a source of information about human environments, and as stimuli in experiments. These uses are sufficiently different that researchers can advocate one while arguing against the other (e.g., Rust & Movshon, 2005). We think that both uses are potentially important for the study of human cognition. However, in arguing for more widespread use of natural images in cognitive science research, we are not arguing against the utility of simple abstract stimuli. These are complementary methods, useful in exploring different kinds of questions about human cognition. In the remainder of this section we highlight three contexts where natural images are particularly valuable: estimating distributions that characterize the environment in which human beings operate, evaluating psychological theories, and taking those theories outside the laboratory and turning them into real applications. These three contexts differ in the way in which natural images are used: the first treats images as data, while the second and third use images either as data or stimuli. In discussing these contexts, we thus also consider how the use of natural images interacts with more traditional experimental methods. Large image databases 5 Estimating distributions A natural question to ask about any aspect of cognition or perception is how much of people’s behavior can be explained by the statistics of their environment. But estimating those statistics can be a challenge. Research on visual perception has made extensive use of images of natural scenes as a source of information about the probability distributions that arise in our natural environment. For example, Geisler et al. (2001) measured the frequencies with which edges co-occurred in images of natural scenes, and showed that the resulting distribution could be use to explain people’s perception of contours. Images are an obvious source of distributional information relevant to vision, but they can also potentially give us clues about other, more cognitive capacities. To return to our running example, consider what information might be extracted relevant to categorization. Current work in computer vision aims to develop automated systems for labeling the contents of images, and large databases of images annotated by humans already exist (e.g., Russell, Torralba, Murphy, & Freeman, 2008). Annotated images carry information about the baserates with which people encounter different categories and the correlations between categories (a source of contextual cues to category membership). When used in combination with tools for extracting high-level visual features (e.g., Donahue et al., 2013), these images also provide a source of hypotheses about the kinds of features people might find highly diagnostic of category membership. Exactly this approach has been used to explore human categorization of scenes (Greene, 2013), and can potentially be pursued for other aspects of categorization.